PATTERN AVOIDANCE AND SMOOTHNESS OF CLOSURES FOR 
ORBITS OF A SYMMETRIC SUBGROUP IN THE FLAG VARIETY 



WILLIAM M. MCGOVERN AND PETER E. TRAPA 

Abstract. We give a pattern avoidance criterion to classify the orbits of Sp(p, C) x Sp(q, C) 
(resp. GL(n, C)) on the flag variety of type Cp{<, (resp. D„) with rationally smooth closure. 
We show that all such orbit closures fiber (with smooth fiber) over a smaller fiag variety, 
and hence are in fact smooth. In addition we prove that the classification is insensitive to 
isogeny. 



Suppose G is a complex connected reductive algebraic group and let 6 denote an involutive 
automorphism of G. Write K for the fixed points of 9, and B for variety of maximal solvable 
subalgebras of the Lie algebra q of G. (Henceforth we call this variety simply the flag variety 
of G.) Then K acts with finitely many orbits on B via the restriction of the adjoint action 
(e.g. |Mat79j l. 

Since we have assumed the ground field is C, ^ arises as the complexification of a Cartan 
involution for a real form of G. The localization theory of Beilinson-Bernstein relates the 
geometry of K orbits on B with the category of Harish-Chandra modules for G^. Meanwhile, 
as a special case, one can consider the setting where is itself a complex Lie group. In 
this case G = x Gr, 6 is the involution that interchanges the two factors, K is the 
diagonal copy of Gr, and the Weyl group of Gr parametrizes the orbits oi K on B (which 
now is two copies of the flag variety Bo for Gr). Intersecting such an orbit with (say) the 
left copy Bo gives an orbit of a Borel subgroup on Bo, that is a Schubert cell. This process 
preserves the fine structure of the singularities of the closures of each kind of orbit, and is 
the geometric underpinning of the equivalence of categories (essentially) between category 
O and a suitable category of Harish-Chandra modules for Gr (e.g. [BorBrySSl ) . 

Thus, roughly speaking, any question which one can ask about Schubert varieties can also 
be posed in the greater generality of K orbits on B, and any relation of the geometry of the 
former with category O can potentially be translated into a relation of the latter with the 
category of Harish-Chandra modules. Some of the deepest results in this direction are due to 
Lusztig and Vogan LusVog83 and Vogan Vog79| , rVog83| , which when taken together give 



an algorithm to compute the local intersection homology (with coefficients in any irreducible 
local system) of any orbit closure. When Gr is a complex group, the algorithm is equivalent 
to that of [KazLus79] for Schubert varieties. 

Our interest here is determining when the closures of K orbits on B are smooth, or 
more generally rationally smooth. Since the latter condition is equivalent to a condition on 
local intersection homology (see the discussion of |KazLus79l Appendix]), the question of 
whether a particular orbit has rationally smooth closure can be answered using the algorithm 
of Vog79| . But it is desirable to have a "closed form" of the answer. For instance, in the case 



that Gr is complex (or, equivalently, the case of Schubert varieties), a closed form answer for 
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smoothness or rational smoothness has been obtained in terms of a kind of pattern avoidance 
for Weyl group element^. 

In the setting of K orbits on new phenomena appear that are not present in the case 
of Schubert varieties. The most obvious difference is that while the definition of Schubert 
varieties is independent of the isogeny class of G, symmetric subgroups (and their orbits on 
B) do indeed depend on isogeny. A typical complication in the latter case may be visualized 
as follows. Suppose G is simply connected but not adjoint, G / G is isogenous to G, and K 
and K are respective symmetric subgroups with the same Lie algebra. Then it frequently 
happens that K is disconnected (while K is connectedjl. Thus there may be two distinct 
orbits Qi and Q2 for K on B whose union forms a single orbit Q for K. Schematically one 
encounters pictures as follows. 



Here Qo is a closed K orbit (which is a also a K orbit) which appears in the closure of both 
K orbits Qi and Q2- The picture indicates that the closure of the the K orbit Qi (or Q2) is 
smooth at Qq. But the K orbit Q, the union of Qi and has closure which isn't smooth 
(or even rationally smooth) at Qo- Moreover, this particular example can "propagate" in 
higher rank schematically as follows. 



This time the closure of the the K orbit Qi (or Q2) is no longer smooth at Qo (but is ratio- 
nally smooth) , but once again the K orbit Q has closure which is not smooth (nor rationally 
smooth) at Qo- Of course none of this is conceptually very complicated, but since, roughly 
speaking, pattern avoidance results are predicated on failure of smoothness propagating uni- 
formly to higher rank, these examples suggest that such results are potentially more subtle 
in the case of K orbits in ways that are not seen for Schubert varieties. This is made clear 
by Example 15.11 

In this paper, we are interesting in understanding nice isogeny-independent pattern avoid- 
ance results. For instance, [McGOTj answers the questions of smoothness and rational 
smoothness for the closures of orbits K = GL(p, C) x GL((7,C) on the flag variety for 
G = GL(p-|-q', C). This is the setting arising from the real form Gjr = U(p, g) of G. After 



There is an extensive literature of geometric and combinatorial results relating pattern avoidance in Weyl 
groups to singularities of Schubert varieties. We do not attempt to recount the history of these results in 
detail here. See [Bil98j . [BilLakOO. Chapter 8], or [BilPosOS] and the extensive references therein. 

^We remark that the notion of isogeny we are considering here differs from isogeny of the corresponding 
real forms. More precisely, if Gh and Ge are the real forms corresponding to symmetric subgroups K and 
K of groups G and G with G simply connected and G a quotient of G by a central subgroup, then of course 
it does not follow that Gb is a quotient of Gh by a central subgroup. (In other words, the relevant central 
subgroup of G need not be defined over R.) For the results of this paper, the latter notion of isogeny of real 
forms is not interesting and will not be considered. 
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reviewing some preliminaries in Section [H we recall the results of [McG07j in Section [2] and 
then prove that they are, in a suitable sense, insensitive to isogeny. 

In the remainder of the paper, we go on to study other classical groups outside of type A. 
In general, as remarked above, the situation potentially depends crucially on issues related 
to isogeny. But, perhaps surprisingly, we find that isogeny essentially plays no role if G is 
of type C (resp. type D) and the Lie algebra of K is 5p(p, C) ©sp(g, C) (resp. 0l(n,C)). 
This setting includes the cases arising from the real groups Sp(p, q) and S0*(2n). The main 
results are Theorems 13.21 and 14.21 They completely settle the question of rational smoothness 
(which, in the end, turns out to be equivalent to smoothness) for K orbits closures in these 
cases. The statements are formulated in terms of a remarkably simple pattern avoidance 
criterion resembling the one discovered in |McG07] . (Roughly speaking there are always 
only seven "bad" patterns to avoid.) 



1. PRELIMINARIES 



In particular examples below, we will need a detailed description of the closure order of 
K orbits on B. We begin by recalling a few features of the general case. These are due to 
Matsuki [Mat79| , Matsuki and Oshima [ MatOshSS] , and Lusztig andVogan LusVog83 , and 
are given a full exposition in Richardson and Springer |RicSpr90j . 



Following the terminology of |RicSpr90 Section 5.1], we first recall the weak closure order 



on K\B. Fix, once and for all, a choice of ^-stable Cartan subalgebra in g, and a choice 
of positive roots A"*" in the full root system A = A(g,f)). Write B for the corresponding 
Borel subgroup of G. For a simple root a € A"*", let Va denote the variety of parabolic 
subalgebras of type a (i.e. those conjugate to the one with roots A+ U {—a}). Write tTq, for 
the projection of B to Va, and define 

(1) Sa - O := the (unique) dense K orbit in 'iT~^{'Ka{0)). 

The weak closure order is generated by relations O < Sq, • O whenever dim(sa • O) = 
dim(C') + 1. In this case, for O' = Sq, ■ O, we write 

O O'. 



Then O C O , but these relations do not generate the full closure order. To obtain all closure 
relations, we must recursively apply the following procedure (implicit in [RicSprQO Theorem 



7.11(vii)], for instance). Whenever a codimension one subdiagram of the form 
(2) Oi 

/3 



a 



4 



is encountered, it must be completed to 

(3) Oi 

04 
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New edges added in this way are represented by dashed hnes in the figures appearing in 
Section [5l Note that this operation must be applied recursively, and thus the solid unlabeled 
edge in the original diagram ([2]) may be dashed as the recursion unfolds. 

We next recall a definition from |RicSpr90 4.7]. Given w € W = W{t),Q), fix a reduced 



expression w = Sq^, • • • SajSa^. For O G K\B, define 

(4) wO = Sa^-{---Sa2-{Sa,-0)). 

It is easy to see this is well-defined independent of the choice of reduced expression. 

The following is our main tool to detect failure of rational smoothness, and is a special 
case of the results of Springer |Spr92| . 

Theorem 1.1. Suppose q contains a Cartan subalgebra which is fixed pointwise by 9. Fix 
O € K\B and a closed orbit O^i € K\0. Consider 

S{0, Od) = {a G A+ I s„ • 0,i + Od and s„ • 0,i C O}. 

If 

(5) #5(0, Oa) > dim(O) - dim(Oez), 
then the closure of O is not rationally smooth. 

The necessary condition for rational smoothness furnished by Theorem 11.11 is a priori 
rather weak; for example, the analogous necessary condition for rational smoothness of 
complex Schubert varieties is far from sufficient, even in low rank. We will see below, 
however, that this condition is both necessary and sufficient for rational smoothness in all 
the cases that we consider. (Even though we make no use of them, we mention that a 
number of other powerful techniques exist for detecting (rational) smoothness; see |Bri99] . 
[BriOOj . and |CarKut03] . for example, and the references therein.) 

2. V{p,q) 

We specialize to the setting of |McG07j before returning to questions of isogeny at the 
end of this section. Fix integers p + q = n, and a signature {p,q) Hermitian form (•, •) on 
an n-dimensional complex vector space Vc. Let G = GL(Vc) (the invertible complex linear 
endomorphisms of Vc with determinant one) and let Gk denote the subgroup of G preserving 
(•,■). Write 9 for the complexification of a Cartan involution of Gu and set K = G^ . Then 
K ~ GL{p, C) X GL{q, C). Choose coordinates so that A"*" = {cj — ej \ i < j}. 

In the present setting, the twisted involutions of |RicSpr90l Section 3] parametrizing K\B 
amount to involutions of Spj^q with signed fixed points of signature {p,q); that is, involutions 
in the symmetric group Sp^q whose fixed points are labeled with signs (either + or — ) so that 
half the number of non-fixed points plus the number of + signs is exactly p (or, equivalently, 
half the number of non-fixed points plus the number of — signs is q). The parametrization 
is arranged to have the following feature. Suppose O is parametrized by an involution with 
signed fixed points whose underlying involution in the symmetric group is a. Then there is 
a representative bo = i)o ® (with l)o ^-stable) with the following property. Write 
for the roots of i)o in g. There is a unique inner automorphism of q carrying f) to f)o and 
A"*" to Aq. Using it we may transport the action of 9 on A J to our fixed system A^. Once 
this is done, the key property is that 

(6) 6'(ei - Cj) = e<^(i) - e<^(j). 
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Let T,±{p,q) denote the set of signed involutions of Sp+q with signature ip,q) as in the 
previous paragraph. For the purposes of formulating pattern avoidance results, we intro- 
duce the following notation. An element 7 G T,±{p,q) will be identified with an n-tuple 
(ci, . . . ,Cn), with each q either a natural number, a +, or a — such that: every natural 
number occurs exactly twice; and the number of distinct natural numbers plus the number 
of + entries is exactly p. For later use in Definition 12.11 below, we say that two such strings 
are equivalent if they have the same signs in the same position and pairs of equal numbers in 
the same positions. (So llH — 22 is equivalent to 22 H — 33 and 55H — 22, for instance.) In any 
event, the correspondence between equivalence classes of such strings and involutions with 
signed fixed points is clear: pairs of equal natural numbers Cj = cj in the string correspond 
to indices i and j interchanged by the involution, and a sign in the string corresponds 
to a label of the fixed points i of the involution. We will generally not distinguish between 
elements of T,±{p,q) and (equivalence classes of) such strings (which, we remark, are called 
"clans" in |Yam97j ^. 

We now turn to an explicit description of the closure order. (It may be helpful to refer 
to Figure 1 when reading the discussion below.) Fix 7 = (ci, . . . ,c„) G T,±{p,q), a simple 
root a = ei — Cj+i G A"*", and recall the action (described before ([6])) of on a determined 
by O-y. Then 

dim {sa -Oy) = 1 + dim(C'^) 

if and only if one of the following conditions hold (using terminology as in |RicSpr90 4.3], 
for instance), 

(a) a is complex (i.e. 9{a) 7^ ±a) and 6a G A+; 

(b) a is imaginary (i.e. 9{a) = a) and noncompact. 

Using ^ for the former condition, and a calculation in U(l, 1) for the latter, these may be 
formulated as the following conditions on positions i and i + 1 in 7: 

(a) Ci and q+i are unequal natural numbers such that j < k where Cj = Cj and Cj+i = 
(and j i and 7^ z + 1); 

(a') Ci is a sign, Cj+i is a number and the entry Ck with Ck = Q+i {k ^ i + 1) satisfies 
i < k; 

number, q+i is a sign, and the other entry Cfc with Cfc = q {k ^ i) satisfies 
k <i + l; 

(b) Ci and Cj+i are opposite signs. 

Thus if i = 2, then 71 = (1,1,2,2) satisfies the first condition above, 72 = (—,+,1,1) 
satisfies the second, 73 = (1, 1, — , +) the third, 74 = (1, +, — , 1) the fourth, and (1, 2, 1, 2) or 
(1, + , 1, — ) satisfies none of them. In each case the dense K orbit in Sa ■ is parametrized 
by 7' = (c^, . . . , c^) G S-|-(p, q) which differs from 7 only in the i and i + 1 entries in each 
respective case as follows 

(a) c- = Cj+i and c-^^ = Cj. 

(b) = c[^i is a natural number. 

So for the examples listed above, we have 7^ = (1,2,1,2), 72 = (— ,1,+,1), and 73 = 
(1,-,1,+), and 7^ = (1,2,2,1). 

The previous paragraph thus gives a complete description of the weak closure order. Af- 
ter applying the recursive procedure given above, one obtains an explicit description of the 
full closure order. In particular, all of the following operations move from one element 
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7 G T,±{p,q) to a higher one in the order: replace a pair of (not necessarily adjacent) op- 
posite signs by a pair of equal numbers; or interchange a number with a sign (again not 
necessarily adjacent to it) so as to move the number farther away from its equal mate in 
the string (and on the same side); or interchange a pair a, b of equal numbers with a to the 
left of b provided that the mate of a lies to the left of the mate of b. Thus (the orbit corre- 
sponding to) (1, +, 1, — ) lies below (1, 2, 1, 2) and (1, +, — , 1), while (1, 2, 1, 3, 2, 3) lies below 
(1, 3, 1, 2, 2, 3) but not below (1, 3, 1, 3, 2, 2). In particular, the closed orbits are parametrized 
by elements of T,±{p,q) consisting only of signs, while the open orbit is parametrized by 
{1,2,..., 2q, +,..., +, 2q - 1, 2q, 2q - 3,2q - 2, . . . ,1,2), with 2p - 2q plus signs, if p > q. 

By similar considerations, one quickly deduces the following dimension formula (as in 
|Yam971 Section 2.3], for instance) for the orbit parametrized by 7 = (ci, . . . , c^). Let 

(7) /(7) = ^2 (j - ^ - #{^ G N I Cs = Q = /c for some s < i < t < j}) . 
Then 

dim{0^) = d{K) + l{-f), 
where d{K) is the dimension of the flag variety for K, namely ^{p{p — 1) + q{q — 1)). 

Definition 2.1. We say that an involution with signed fixed points (ci, . . . , c„) G T,±(p, q) 
includes the pattern {di, . . . ,dm) if there are indices ii < ■ ■ ■ < so that the (possibly 
shorter) string (cj^ , . . . , Cj^) is equivalent to (di, . . . , dm)- We say that 7 avoids (di, . . . , dm) 
if it does not include it. For instance, (1, 1, 2, +, 3, 2, — , 3): contains the pattern (1, 1, 2, 2) 
(by considering ii = 1,12 = 2, is = 3, and 14 = 6), contains (1,2,1,2) (by considering ii = 
3, 12 = 5, is = 6, and = 8), and contains (1, +, 1, — ) (by considering ii = 3,12 = 4, is = 6, 
and i4 = 7); but avoids (1, +, -, 1), (1, +, +, 1), and (1, 2, 2, 1). 

Here is the main result from |McG07j . 

Theorem 2.2. Fix 7 G T,±{p,q), an involution in Sp^q with signed fixed points of signature 
{p, q) (as defined above). If ^ includes one of the patterns (1, +, — , 1), (1, — , +, 1), (1, 2, 1, 2), 
(1, +, 2, 2, 1), (1, -, 2, 2, 1), (1, 2, 2, +, 1), or (1, 2, 2, -, 1), then the closure of is not ratio- 
nally smooth. In all other cases, has smooth closure. In particular, an orbit has smooth 
closure if and only if it has rationally smooth closure, or if and only if the condition of 
Theorem Ii. il fails for some closed orbit below it. 

Next we turn to issues of isogeny. First, we switch notation and consider the simply 
connected simple group G = SL(n,C). Since the center of GL(n, C) acts trivially on B, the 
above discussion applies without change for symmetric subgroup K = S(GL(p, C) xGL(g, C)) 
of G. Now let G be a quotient of G by a subgroup F < Z/n of the center of G. Let K be 
the symmetric subgroup of G with Lie algebra s(0[(^j, C) © 0((g, C)). Then the orbits of K 
on B coincide with those of K except in one case: if n = 2m, p = q = m, and F contains the 
index m subgroup of Z/2m. In this case, K has two connected components, and indeed K 
orbits can be disconnected. More precisely, K orbits are parametrized by Ti±{p,q) modulo 
the equivalence relation generated by 7 ~ —7, where —7 is obtained from 7 by reversing all 
signs of fixed points. (So, for instance, (1, +, — , 1) is equivalent to (1, — , +, 1), but (1, 2, 1, 2) 
is equivalent to only itself.) Equivalence classes thus have either one or two elements. In 
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general, if 7 denotes such a class, the K orbit parametrized by 7 breaks into K orbits as 
(8) 0- = Oj\J O-^ 

which is disconnected if 7 7^ —7. So the subtleties alluded to in the introduction could 
potentially come into play. In fact, they do not, and one way to see this is to examine the 
proof of Theorem 12.21 It consists of two steps: first proving that if 7 contains one of the 
three indicated patterns, then it is not rationally smooth; and, second, proving that the 
remaining orbits are smooth. The first is done by finding a suitable closed orbit Od in the 
closure of so that Theorem 11.11 applies. It turns out that the identical argument can 
be carried out for K orbits; that is, roughly speaking. Springer's criterion is insensitive to 
isogeny in this case. (This need not always be true — see Example 15.11 ) The second step 
can also be carried out in an analogous manner which, once again, is a special feature of this 
case. In the end, one concludes Theorem \2.S\ holds for the orbits of any symmetric subgroup 
K of G (isogenous to SL(n,C)j with Lie algebra 5(0[(p, C) ©0^(^i',' 



We conclude this section by recalling (as in [RicSprOO Section 2]) the natural action of 



W = W{\),q) on the set of twisted involutions parametrizing K orbits on B. (We will need 
this for formulating some results in Section HI) On the level of Ti±{p,q) we write w x j for 
the action of w on 7. Explicitly, it amounts to the obvious action of the symmetric group: 
the w{i)th. component of (the string parametrizing) if x 7 is simply the ith component of 
(the string parametrizing) 7. 



3. Sp{p,q) 

We now turn our attention to Gk = Sp(p, q), a real form of G = Sp(2n, C), again deferring 
isogeny questions to the end of this section. We start with a brief outline of our strategy. 
As the definition recalled below makes evident, Gr is a subgroup of GJj = U(2p, 2q). In this 
section, we will try (as much as possible) to reduce the study of K orbits on B, the flag 
variety for G, to corresponding results in the previous section for K' ~ GL(2p, C) x GL(2g, C) 
orbits on B', the flag variety for GL(2p + 2g, C). More precisely, B naturally includes into B' 
and (for appropriate choices of the Cartan involutions in question) a K' orbit on B' either 
meets B in a single K orbit, or else not at all. Thus the orbits of K on B are parametrized 
by a subset of the involutions with signed fixed points T,±{2p, 2q) introduced above. In fact, 
we quickly check below that the closure order on K orbits on B is simply the appropriate 
restriction of the closure order of K' orbits on B'. Thus the simplest result that one could 
hope for is this: the closure of an orbit for K on B has (rationally) smooth closure if and 
only if the corresponding K' orbit B' which it meets has (rationally) smooth closure (and 
recall these latter orbits have been classified in Theorem 12.21) . This is in fact the content of 
Theorem 13 . 2 1 b elow . apart from an easily stated exception treated in Lemma l3.1[ 

It is worth remarking that since (up to issues of isogeny) any classical real group outside 
of type A is a subgroup of an appropriate U(p, g), one could attempt to mimic the strategy 
of the previous paragraph for any such group. In the next section, we do so for S0*(2n) and 
obtain similar results. But for the other classical groups, complications arise; see Example 

EH 

We return to the details of the case of Sp{p,q). Let IHI denote the quaternions equipped 
with the standard bar operation a + hi + cj + dk = a — hi — cj — dk. Embed C in H, as 
usual, as elements of the form a + hi and write the corresponding isomorphism H ~ as 
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z = A{z) + B(z)j. Let (•, •) denote a signature {p,q) sesquilinear form on an n-dimensional 
quaternionic (left) vector space Vm- Define 

(,.)' = Ao(,.) 

and 

(.,.)" = Bo (,.). 

Then (•,•)' is a nondegenerate Hermitian form of signature {2p,2q) on the underlying 2n- 
dimensional complex vector space Vc and (•,•)" is a nondegenerate alternating form on Vq- 
Let Gm = Sp{p,q) denote the subgroup of GL(lii) (the group of invertible left H-linear 
endomorphisms of \%.) preserving (•, •) and view, as we may, Gk as subgroup of GL(Vc). 
Let G'^ denote the subgroup of G' := GL(Vc) preserving (•,■)', and finally let G denote the 
subgroup of GL(Vc) preserving (•,•)". Then Gm. is a real form of G ~ Sp(2n, C), G'^ ~ 
U(2p,2g), and 

Gr = Gjj n G. 

We adopt the notation of Section [2] for GJ^, adding a prime everywhere as appropriate. 
Any Cartan involution for Gjj restricts to one for Gk- If we write 9' for the corresponding 
complexified involution of G' and 6 for its restriction to G, we naturally have 

Sp(2p,C) X Sp(2g,C) K := G^ < K' := {G'f ~ GL(2p,C) x GL(2g,C). 

Recall the natural inclusion of the flag variety B for G into B', the flag variety for G'. By 
the remarks above, in order to classify K orbits on B it suffices to determine which orbits O' 
meet B nontrivially. If 7' = (c'^, . . . , C2„) G T,±{2p, 2q) (with notation as in the last section), 
we say that it is symmetric if 

(i) If the entry is a sign, then C2„_,_^_j is the same sign. 

(ii) If c- = c'j are natural numbers, then j 7^ 2n + 1 — i and C2„^_;^_j = C2„^_;^_j. 

We write S^™(2p, 2g) for the subset of symmetric elements in T,±{2p,2q). Then the K' 
orbit Oy meets B if and only if 7' is symmetric ([MfliQshSSj, |Yam97t Section 4.3]), and 
thus S!^™(2p, 2q) parametrizes K orbits on B. Our next task is to describe the closure order 
explicitly. 

Fix, as in Section [2l a ^'-stable Cartan subalgebra i)' , a choice of positive roots, and 
(in appropriate coordinates) write (A')+ = {e^ — e'j \ i < j}. Then f) := q Ci i}' is a 9- 
stable Cartan subalgebra of q. Restriction defines a positive system of roots of f) in g, 
A = {2ei I 1 < i < n} L) {ei ± Cj 1 < i < j < n}. Let B denote the corresponding Borel 
subgroup of G. Fix a simple root a in A"*" and let S'{a) the set of roots in (A')"*" which 
restrict to a. Concretely, if a = Cj — Cj+j, then S'{a) = {a', a"} with a' = e[ — e[_^_^ and 

~ ^2n-i ~ ^2n-i+v if a = 2ei, then S'{a) = {a'} where a' = e'^ — e'^j^^. 

Given a simple root a G A+ and O in K\B, define Sa ■ O as in ([1]). For /3 G (A')"*" and 
O' in K'\0', define s'^ ■ O' similarly. The parametrization of K and K' orbits satisfies the 
following key geometric compatibility condition (which follows easily from unraveling the 
definitions). Given any symmetric 7, let O-y denote the corresponding K orbit on B and O'^ 
the corresponding K' orbits on B'. Then 

(9) dim {sa-Oy) = l + dim(O^) 
if and only if 

(10) dim (4 . O;) = 1 + dim(0;) 
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for some (equivalently, any) root (3 £ S'{a). Moreover, in this case, the the dense K' orbit 



m 



s. 



n 

intersected with B is the dense K orbit in 

(12) 7r-i(7r,(0^)). 

Note that if the product in (jlip has more than one term, the order of the terms does not 
matter since, in this case, the roots in S'{a) are orthogonal. 

Using these facts, it is a simple matter to write down the weak closure order on the level 
of S!^™(2p, 2g) from the description of the weak closure order described in the previous 
section. (This is done in [Yam971 4.4], for instance.) By examining the recursive procedure 
to generate the full closure order from the weak order, one concludes that the closure order 
of K orbits on the level S!^™(2p, 2g) is simply the restriction of the order on Ti±{2p,2q) 
given in the previous section. The dimension of the orbit parametrized by 7 G S!^™(2p, 2q) 
is also easy to read off, and is given by 

dim(O^) = d{K) + (1/2) (/(7) + {t e N | = q G N with s <n <t <2n + l - s}) 

where d{K) is the dimension of the flag variety for namely + q'^, and is as in ([7]). 
In particular, closed orbits are once again parametrized by elements consisting only of signs, 
while the open orbit is parametrized by 

(13) 7o(p,(?) := (1,2,..., 2(7, +,...,+,2(7-1,2*7, 2(7-3,2*7-2,..., 1,2) 

with p — q plus signs if p > (7, and similarly if (7 > p. See Figure 2 for a detailed example. 

We need some notation for the next result. If 71 = (ci, . . . ,Ck) represents an involution 
with signed fixed points, then let jI be the string with the coordinates of 7 in reverse order 
but with each pair of equal numbers changed to a different pair of numbers. For example, 
if 71 = (1, 2, +, 1, 2, — ), then 7[ = (— , 3, 4, +, 3, 4). For any such 71, the concatenation 
7 := (71,71) is symmetric, so may be viewed as an element of S!^™(2p, 2(7). 

Lemma 3.1. Fix 7 G S!^™(2p, 2q). Suppose that there are integers p' + r = p and q' + s = q 
so that 7 can be written as the concatenation 

(7i>7o(p',g')>7i) 

with 71 € S-|-(r, s) such that 71 avoids the bad patterns of Theorem \2JA Then the closure of 
is a fiber bundle with smooth fiber over a partial flag variety for K and hence is smooth. 
The fiber is isomorphic to the product of the flag variety for Sp(2p'+2(7', C) and the closure 
of the GL(r, C) x GL(s,C) orbit parametrized by (as in Section\^. 

Proof. We start with a general observation. Suppose 7 is any element of Ti±{p^q) which 
can be written as a concatenation (7i,7o,7i) where 70 € 'E±{p',q') and 71 € S^™(2r, 2s). 
Set n' = p'+q', n" = r+s. Then there is a ^-stable parabolic subgroup Q = LU, unique up 
to K conjugacy, containing B with L ~ Sp(n',C) x GL(n",C) and 

n L ~ Sp(p', C) X Sp{q', C) X GL(r, C) x GL(s, C). 

The assumption that 7 = (71, 7o! 7i) implies that the image of under projection vr-p from 
B c:^ G/B to V — G/Q \s the closed K orbit of the identity coset eQ and that the fiber of 
the restriction of vr-p to is a single orbit oi Lr\ K on Bl, the flag variety for L. The fiber 
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is isomorphic to the product of the Sp(p',C) x Sp(g',C) orbit parametrized by 70 with the 
orbit of GL(r, C) x GL(s,C) parametrized by 71. Meanwhile the K orbit of eQ is closed, 
and hence isomorphic to a partial flag variety for K. The closure of is thus a fiber 
bundle over a partial flag variety for K whose fiber is the closure of the product of the orbits 
parametrized by 70 and 71. If we impose the additional hypothesis that 70 = 7o(p', q') and 
71 avoids the patterns given in the lemma, then we conclude from Theorem 12.21 that the 
fiber is indeed smooth, and the remainder of the lemma follows. □ 

Theorem 3.2. Fix 7 € Yf^™[2p,2q), a symmetric involution of S2p+2q with signed fixed 
point of signature {2p,2q) (as defined above). Suppose 7 is not of the form treated by Lemma 
\3.1\ and, further, that 7 includes one of the bad patterns of Theorem \2.2l Then the closure 
of does not have rationally smooth closure. 

In all other cases, the closure of is a fiber bundle with smooth fiber over a partial flag 
variety for K, and hence is smooth. In this case, there are integers r + s + n' = p + q so 
that the fiber is isomorphic to the product of a flag variety for Sp(2n',C) and the closure of 
an orbit of GL(r, C) x GL(s, C) on the flag variety for GL(r+s, C). 

In particular, has rationally smooth closure if and only if it has smooth closure, or if 
and only if the condition of Theorem fails for every closed orbit below it. 

Proof. If 7 is of the form treated by Lemma [3. H there is nothing to prove. So suppose this 
is not the case and that 7 avoids the patterns given in the theorem. Then 7 takes the form 
(71, 7[) with 71 € T,±{p,q) which avoids the same patterns. Thus Lemma |3 . II applies (with 
p' = q' = 0) to give the required assertion. 

Suppose 7 contains one of the patterns of the theorem. Then there are various geometric 
ways to deduce the failure of the closure of to be rationally smooth by embedding 
the corresponding failure for the smaller rank group where, roughly speaking, the pattern 
resides. (The exceptions of Lemma l3.ll shows that some care is required.) A concise (and 
convenient) way to organize the geometric reduction is to use Theorem II. H and this is how 
we shall proceed. 

In the present context a root — ej with i < j is noncompact imaginary for an orbit Od 
parametrized cl if the ith and jth. coordinates of cl are opposite signs and the root Cj + ej 
is noncompact imaginary if the ith and 2n + 1 — jth. coordinates are opposite signs. (The 
root 2ej is never noncompact imaginary.) Then if a = Cj — Cj, Sa ■ O^i is parametrized by the 
element obtained from cl by replacing its zth, jth and 2n + l — jth, 2n + 1 — ith coordinates 
by different pairs of equal numbers; if instead a = Ci + ej , the new element is obtained by 
replacing the ith, 2n + l— j'th and jth, 2n + l — ith coordinates by two different pairs of equal 
numbers. As we remarked above, closed orbits are parametrized by elements consisting of 
only signs. Since we have given the closure order and dimension formula above, the criterion 
of (l5|) becomes very explicit, and we may apply it directly as follows. 

Fix a 7 = (ci,...,C2n) G S±™(2j>, 2^) including one of the bad patterns and not of 
the form treated by Lemma 13.11 We first produce a suitable closed orbit Od lying below 
O-y. Look first at the natural numbers occurring twice among c\ . . . , Cn . Replace the first 
occurrence of all such numbers by + and the second by — . Then look at all the natural 
numbers occurring just once among ci, . . . , c„. Whenever = C2n+i-j is a natural number 
and i < j < n (so that Cj = C2n+i-i is also a natural number), replace by + and Cj by 
— . Finally, replace Cn+i, . . • , C2n by signs in such a way that the result cl = (di, . . . , cl2n) is 
symmetric. 



PATTERN AVOIDANCE AND SMOOTHNESS 



11 



Assume for the moment that there are no indices i < j < n with q and cj equal natural 
numbers. Enumerate the indices i < n with Cj and C2n+i-j equal natural numbers for 
some J < n as ii < • • • < i2m- Define a new element 7' = (c'l, . . . ,C2„) by decreeing that 
(4'--->4+2m-i) = (l,--->2m), that {c'^^^^_^2m-i)~h^ ■ ■ ■ ^^M-h) = (2m -1,2m, 2m- 
3, 2m — 2, . . . , 1, 2), and finally that there be as many + signs among (c'l, . . . , c^) as among 
(ci, . . . , Cn) and similarly for — signs. Then 7' parametrizes an orbit Cy whose dimension 
is at least that of O^, but for which the left-hand side of ([5]) for Oy (and Od) is no larger 
than it is for (and Od)- Thus if either Springer's criterion implies that the closure of 
Oy fails to be rationally smooth, or Oy has larger dimension than O^, then the closure 
of O-y is rationally singular. Note that both sides of ([5]) are unaffected if we replace 7' 
by the new (smaller) element obtained by deleting all the initial signs c[,...,c[ _i and 
terminal signs C2„_j^_,_2, • • • ,C2„ from 7', and replace cl by the element obtained by deleting 
the corresponding initial and terminal entries from cl. So we may assume ii = 1. 

We argue inductively based on the number of distinct natural numbers k appearing in 7' 
as follows. We will reduce our analysis to the case of A: = 2, so assume first that k > 2. 
Consider the new element 7" (of smaller size) obtained from 7' first by deleting the initial 
entries c[,...,c'^^ and terminal entries C2„+i_i2i • • • j C2„) and then deleting all the initial 
and terminal signs from the result. Let cl" denote the element obtained by deleting the 
corresponding entries from cl. Then in passing from Oy to Oy (and from Od to Od"), 
the difference of the left- and right-hand sides of ([5]) will strictly increase in all cases unless 
Z2 = 2, in which case the difference will remain the same. Continuing this procedure, we 
are thus led to investigating the "base case" of the form 7" = (1, 2, £3, ... , e„, e„,, . . . , €3, 1, 2) 
and cl" = (+, — , 63, . . . , En, en, ... , £3, — , +)• There are two possibilities. If the signs ej are 
not all the same, one checks directly that ([5]) holds for the base case. Since each of the steps 
leading to the base case weakly increases the difference of the left- and right-hand sides of 
dS]), one concludes that ([5]) holds for Oy and Od, hence for O-y and Od, and hence the 
closure of Oy is not rationally smooth, as claimed. The other possibility is that all of the 
signs in the base case are the same. In this case, the two sides of ^ corresponding to the 
base case are actually equal. Again since each step in the reduction weakly increases the 
difference of the left- and right-hand sides of ([5]), the inequality in ^ holds for O'^ and Od 
(and hence the closure of Oj fails to be rationally smooth) except possibly in just one case, 
namely when each step reducing to the base case does not increase the difference of the left- 
and right-hand sides, and when the base case turns out to have all signs the same. But the 
only way this can happen is if 7' is of the form Joip^q') (possibly flanked by a number of 
signs). If this is the case, the construction of 7' shows that 7 must be of the same form, up 
to a permutation of the natural number entries. Since we have assume that 7 is not of the 
form treated by Lemma 3.1, one checks that the initial passage from 7 to 7' indeed increases 
the difference between the left- and right-hand sides of (l5|). So once again the closure of O-y 
is not rationally smooth, as desired. 

Finally return to the case where there are indices i < j < n with q and cj equal natural 
numbers in 7, and replace q, Cj for every such pair i < j hy signs as in the definition of cl 
above (and define C2n+i-i,C2n+i-j so that the resulting element is symmetric). We obtain 
an element parametrizing an orbit that either does not contain any of the bad patterns, or 
whose closure fails to be rationally smooth by the above argument. Changing the iih and jth 
coordinates of this element back to q , cj (and similarly for the 2n + 1 — ith and 2n + 1 — jth 
coordinates), we find that the dimension of the corresponding orbit increases, but so too 
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does the left-hand side of ([5]) by at least the same number, and by a larger number if two 
of the signs between Cj and Cj are different. We conclude in all cases that the closure of O-y 
is not rationally smooth, as desired. □ 

Finally we must consider the situation for the symmetric subgroup K G = PSp(2n, C) 
with Lie algebra 5p(2p, C) x 5p{2q,C). Then K is connected (and its orbits on B coincide 
with those of K) unless p = q, in which case K has two components. In this case, orbits of K 
are parametrized by equivalence classes in S!^™(2p, 2q) generated for the relation generated 
by 7 ~ —7. Just as in the case discussed at the end of Section [21 if 7 denotes such a class, 
the K orbit parametrized by 7 breaks into K orbits as 

which is disconnected if 7 7^ —7. Once again one may retrace the steps of the proof of 
Theorem 13.21 to show that no complications arise. One thus concludes: Theorem \3.S\ holds 
for the orbits of any symmetric subgroup K of G (isogenous to Sp{2p + 2q,C)) with Lie 
algebra 5p{2p,C) ©sp(2g,C). 

4. S0*(2n) 

We follow the same strategy as outlined at the beginning of Section [3l Let (■, •) denote a 
nondegenerate skew sesquilinear form on an n-dimensional quaternionic vector space V, so 
that 

{u,v) = -{v,u). 

As in the previous section, set 

(.,.)' = Ao(,.) 

and 

(•,.)" = Bo (,.). 

Then (•,•)' is a nondegenerate Hermitian form of signature (n, n) on the underlying 2n- 
dimensional complex vector space Vq and (•,•)" is a nondegenerate symmetric form on Vq. 
Let Gr = S0*(2n) denote the subgroup of GL(ViHi) preserving (•, •) and view as subgroup 
of GL(Vc). Let G'jg^ denote the subgroup of G' := GL(Vc) preserving (•, •)', and let G± denote 
the subgroup of GL(Vc) preserving (•, •)". Then G± ~ 0(2n,C), G'^ ~ U(n,n), and 

Gr = G'^n G±. 

(In fact, every element of Gm. has determinant one, so Gr is indeed a real form of the 
connected algebraic group G ~ S0(2n,C) consisting of determinant one elements in G±.) 
Fixing compatible involutions 6 and 6' as in Section [3l we naturally have 

GL(n,C) ■.= G^ <K' := {G'f ~ GL(n,C) x GL(n,C). 

Let B denote the flag variety for G and B' the flag variety for G'. Once again, there is a 
natural inclusion B' C B, and if O' is an orbit of K' on B', then its intersection with B is 
either empty or consists of a single orbit of K on B. If 7' = (c'^ , . . . , C2„) G ^±{n, n), we say 
that it is antisymmetric if 

(i) If the entry is a sign, then C2n+i-i is the opposite sign. 

(ii) If c- = c'j are natural numbers, then j 7^ 2n + 1 — i and C2„_,_^_j = C2„_^_l_j. 

(iii) Among the entries c'^^, . . . , c^, the number of + signs plus the number of pairs of equal 
natural numbers is even. 
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Write S^^™(n, n) for the set of such antisymmetric elements. Notice there is a choice of sign 
in (iii) above. In Section [2l when p = q = n, the parametrization of orbits can obviously 
be twisted by an outer automorphism of U(n, n), the effect of which is to change all signs. 
Possibly after twisting the parametrization of Section [21 the K' orbit Oy meets B if and 
only if 7' is antisymmetric. Thus S^^™(n, n) parametrizes K orbits on B |MatOsh88j . (It 
may be helpful to refer to Figures 3 and 4 in the course of the discussion below.) 

Fix a ^'-stable Cartan subalgebra i)', a choice of positive roots, and write (A')'^ = {e^ — 
e'j \ i < j}. Then := gPlf)' is a 6'-stable Cartan subalgebra of g. Choose a positive subset of 
roots of f) in q, in appropriate coordinates write A"*" = {(cj it ej) \ i < j}, and let B denote 
the corresponding Borel subgroup of G. Fix a simple root a in A"*" and let S'{a) the set of 
roots in (A')+ which restrict to a. For instance, if a = Cj — Cj+j, then S'{a) = {a' , a"} with 
a' = e^-e^+i and a" = e^„_^ - e'^^_i^^. 

Adopt notation analogous to that around Equation Q, fix a = — ej+i G A+, and 
an antisymmetric element 7 of signature (n,n). Then once again we have the identical 
conclusions of Equations ([9|)- (fT2]) . The situation for a = e„,_i + e„ is more subtle, however. 
Set (y.n—1 — Cn— 1 ~ fix an antisymmetric element 7 of signature {n,n), and let s' denote 
the reflection in the simple root — e'^_^_i in (A')"*". Recall the action described at the end 
of Section [2j Then 

(14) dim {sa-0^) = l + dim(O^) 
if and only if 

(15) dim(4.0;,^^) =l + dim(0;,^^) 

for some (equivalently, any) root (3 £ 5'(a„_i). Moreover, in this case, if O'^ is the dense 
K' orbit in s'^ ■ Oiy, then O'^/^g intersected with B is the dense K orbit in Sa • O-y. Using 
this, one may deduce that the closure order of i^-orbits on the level of Sj-'^ (ti, ?t.) is once 
again the restriction of the order on Y^±(n, n) given in Section [2j The dimension of the orbit 
parametrized by a 7 € S^^™(n,n) is given by 

dim(O^) = d{K) + (1/2) (/(7) - {t G N | = Q G N with s < n < t < 2n + 1 - s}) 

where d{K) is the dimension of the flag variety for namely ^n(n — 1), and /(7) is as in 
d?]). Closed orbits are once again parametrized by elements consisting only of signs, while 
the open orbit is parametrized by 

(16) 7o(n, n) := (1, 2, . . . , 2m - 1, 2m, 2m - I, 2m, ... , 1, 2), 
if n = 2m is even and 

(17) 7o(n, n) := (1,2,..., 2m, -, +, 2m - 1, 2m, ... , 1, 2), 

if n = 2m + 1 is odd. We let ±70(71, n) denote either 70 (n.n) or the element obtained from 
inverting the signs in 7o(n,n) (which differs from 7o(n.n) only if n is odd and, in this case, 
no longer satisfies condition (iii) above). 

We record two more operations on S^'^™(?7-,n). Fix 7 G S^^™(n,n) and let s' be as 
above. Let 7' denote the element obtained by changing all signs in s' x 7 (described at 
the end of Section [2]). Then exactly one element of {7',^' x 7'} is antisymmetric. Let t(7) 
denote this element. Thus r is an involution on S^^™(n, n), and hence can be interpreted 
as an involution of the set of K orbits on B. It coincides with the action of an outer 
automorphism of G. (As an example, r corresponds to the obvious symmetry in Figures 
3 and 4 below.) Finally, if 7 G Ti±{r,s), denote by 7"^ the element obtained from 7 by 
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reversing its coordinates, changing all of its signs, and replacing every pair of equal natural 
numbers by a different pair of equal natural numbers. Then the concatenation (7, 7"'') is 
antisymmetric and so may be viewed as an element of S^^™(r + s, r + s) 

Lemma 4.1. Fix j € S^'^™(n,n) 

(a) Suppose that there are integers r + s + n' = n so that 7 can be written as the 
concatenation 

(7i,±7o(n',n'),7r'') 

where ±70(71.', n') is defined after (jl7p . 71 G $]-|-(r, s) which avoids the patterns of 
Theorem \2.2l and 'j^ '" is defined as above. Then the closure of is a fiber bundle 
with smooth fiber over a partial flag variety for K , and hence is smooth. The fiber 
is isomorphic to the product of the flag variety for S0(2n',C) and the closure of the 
GL(r, C) X GL(s,C) orbit parametrized by 71 (as in Section\^. 

(b) Suppose n = 2m is even and that 7 can be written as 

(l,7l,2,l,7^^2), 

where 71 G 'E±{r — l.s — 1) , r + s = m, avoids the patterns of Theorem \2.^ Then the 
closure of is a fiber bundle with smooth fiber over a partial flag variety for K, and 
hence is smooth. The fiber is isomorphic to the closure of the GL(r, C) x GL(s,C) 
orbit parametrized by the element (l,7i,l) G Ti±{r,s). 

Proof. Part (a) is proved much the same way as Lemma [3 .H and we omit the details. For (b), 
note that the closure of the orbit parametrized by 7 is isomorphic to the one parametrized 
by the outer automorphism conjugate t(7) described above. But if 7 has the form indicated 
in (b), then t(7) has the form indicated in (a). Thus (b) follows from (a). □ 

Theorem 4.2. Fix j G S^-^™(n,n), an antisymmetric involution of S2n with signed fixed 
points of signature {n,n) (as defined above). Suppose 7 is not of the form treated by Lemma 
\4.1\ and, further, that 'y includes one of the patterns of Theorem \2.SX Then the closure of 
does not have rationally smooth closure. 

In all other cases, the closure of O-y is a fiber bundle with smooth fiber over a partial flag 
variety for K , and hence is smooth. In this case, there are integers r -\- s + n' = n so that 
the fiber is isomorphic to the product of the flag variety for S0(2n',C) with the closure of 
an orbit of GL(r) x GL(s, C) on the flag variety for GL(r+s, C) with r+s = n. 

In particular, has rationally smooth closure if and only if it has smooth closure, or if 
and only if the condition of Theorem \l.l\ fails for some closed orbit below it. 

Proof. This is very similar to the proof of Theorem 13.21 We omit the details. □ 

Finally we turn to issues of isogeny. First assume n is odd, so that the center of 
Spin(2n,C) is Z/4. So the three complex groups G to consider are Spin(2n, C), S0(2n,C), 
and PSpin(2n, C). It turns out that the symmetric subgroup K with Lie algebra g[(n, C) is 
connected in each of these cases, and that the orbits of i^T on ,B are insensitive to isogeny. 
So Theorem 14.21 applies without change. 

The case of n = 2m even is more interesting since the center of Spin(2n, C) is Z/2 x Z/2. 
Write S0(2n, C) for the quotient by the diagonal Z/2, S0'(2n, C) ~ S0"(2n, C) for either of 
quotients by an off-diagonal Z/2, and PSpin(2n,C) for the adjoint group. Write K^c, K, K', 
and Kg^^ for the corresponding symmetric subgroups with Lie algebra 0l(n, C). The orbits 
of Ksc and K on B always coincide, so are treated by Theorem 14.21 If m is even, these orbits 
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also coincide with the orbits of K' on B. If m is odd, the orbits of K' instead coincide with 
the orbits of -fCad on B. Finally, the orbits of K^^ are parametrized by equivalence classes in 
n) for the relation generated by 7 ''"(7) for the involution of S^^™(n, n) described 
above. If 7 denotes such a class, the K orbit O— parametrized by 7 breaks into K orbits as 

which is disconnected if 7 7^ ''"(7)- Nonetheless the (omitted) proof of Theorem 14.21 goes 
through unchanged to establish that the statement of the theorem requires no modification 
in this case. 

The conclusion of the discussion is: Theorem \4-'A holds for the orbits of any symmetric 
subgroup K of G (isogenous to Spin(2n, C)J with Lie algebra g((n,C). 

5. EXAMPLES 

We conclude with several examples of the closure order of K orbits on B. Vertices in 
the diagrams below correspond to orbits, and orbits of the same dimension appear on the 
same column (or, in the case of Figure 2, row). Recall that the closure order is generated 
by relations in codimension one; so each diagram need only keep track of such relations. 
Dashed edges correspond to relations not present in the weak closure order, as described at 
the beginning of Section [2j 

Figure 1 corresponds to the case of SU(2,2); that is, the case where G = SL(4, C), 
K = S(GL(2,C) X GL(2,C)), a nd B consist s of complete flags in C^. (Without the dashed 
edges and boxed vertices this is [MatOsh88| Figure 7].) The labeling of simple roots is given 
by 

12 3 



The boxed vertices correspond to orbits with nonsmooth closures according to the pattern 
criterion given in Theorem 12. 2i Suppose now G is a nontrivial quotient of G by a central 
subgroup, and let K be the corresponding symmetric subgroup of G. According to the 
discussion at the end of Section [2l quotienting Figure 1 by the obvious Z/2 symmetry gives 
the corresponding picture for K orbits on B. 

Consider next the case of Sp(2,2) where G = Sp(8,C) and K = Sp(4,C) x Sp(4,C). 
According to the details of Section [3l there are 42 orbits for Sp(2,2), too many to fit in a 
reasonable diagram. Instead consider G = PSp(8, C) = G/F where F is the two element 
center of G. Then K = K/F is a symmetric subgroup of G corresponding to PSp(2, 2). The 
orbits of K on the flag variety are given in Figure 2. (This graph, without the dashed edges 
and boxed vertices, is [MatOshSS^ Figure 15].) Simple roots are labeled as 

12 3 4 
• • • • 

According to the discussion at the end of Section [3l an orbit labeled by an element of 
S!^™(2, 2) which contain signs is the (disconnected) union of two K orbits. (Nonetheless, 
such an orbit has smooth closure if and only if each connected component has smooth clo- 
sure.) Boxed vertices in Figure 2 correspond to K orbits with nonsmooth closures according 
to the criterion of Theorem 13. 2i 
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Figure 3 gives the case of S0*(6), i. e. = GL(3,C) orbits acting on the flag variety for 
so(6, C) ~ s[(4, C). (This diagram is [MatOshSSj Figure 20].) Simple roots are labeled as 

1 2 3 
• • • 



Since S0*(6) is a quotient of SU(3, 1) by the subgroup Z/2 (defined over M) of the center 
Z/4 of Spin(6, C) = SL(4, C), either Theorem 12.21 (and the discussion at the end of Section 
E]) or Theorem 14.21 applies to give that all orbits have smooth closure in this case. 

Finally, Figure 4 gives the case of S0*(8), i.e. GL(4, C) orbits on the flag variety for 
so(8, C). (This diagram without the dashed edges or boxed vertices is [MatOshSSt Figure 
19].) Simple roots are labeled as 



1 




To conserve space, we introduce the following shorthand (as in |MatQsh88| ). The eight sym- 
bols of an element 7 S S^^^™(4, 4) are compressed to just four. The signs in the compressed 
symbol match those in the flrst four coordinates of 7; a pair of numbers in 7 in positions 
i and j with 1 < i < j < 4 is represented by a lower-case letter in positions i and j of 
the compressed symbol; and a pair of numbers in 7 in positions i < 4 and 9 — j > 4, is 
represented by an upper-case letter in positions i and j of the compressed symbol. So, for 
instance, the 7 = 1 + -12 + -1 becomes a + -a; +12 + -12- becomes +AA+; 12123434 
becomes abab; and 12341234 becomes ABBA. With this convention, the boxed vertices 
correspond to orbits with nonsmooth closures according to the condition of Theorem 14. 2i 
Isogeny considerations amount (possibly) to folding the flgure by the obvious symmetry, as 
explained at the end of Section HI 



Example 5.1. We conclude with some examples where isogeny considerations necessarily 
make formulating pattern avoidance results more complicated and less uniform. Let G = 
Sp(2n,C),G = PSp(2n,C) = G/F, K = GL(n,C), and K = K/F. By considerations 
similar to those treated in Sectional orbits of -fC on jB are parametrized by E^-^™(2n), the 
union over all p+g = n of antisymmetric elements in S±(p, q). Orbits of K are parametrized 
by equivalence classes in S^-^™(2n) for the relation generated by 7 ~ —7 and the obvious 
version of ^ holds. This time, however, the relationship between the (rational) smoothness 
of the closure of and the (rational) smoothness of the closures of its connected components 
is a little complicated. For instance, let 72 = H — 1, 73 = +H — 1— , and 74 = 112 H — 233. 
Then each 7j contains (in the sense of Deflnition I2.ip the pattern 1 H — 1. The K orbits 
O^. each have closures which are smooth but not rationally smooth. Meanwhile the K orbit 
O^^ has closure which is not rationally smooth, has closure which is rationally smooth 
(but not smooth), and has closure which once again is not rationally smooth. (One may 
prove the rational smoothness assertions using Theorem ll.lt so, in particular, the criterion of 
the theorem is sensitive to isogeny.) Further calculation suggest a relatively simple pattern 
avoidance criterion for (rational) smoothness of K orbit closures may exist, but formulating 
such a result for K orbit closures is messier. The situation is similarly complicated for 
SO(p, g) (and even more so if p + g is divisible by four). This example suggests that it is 
perhaps reasonable to assume that G is simply connected (and thus K is connected) when 
formulating pattern avoidance results in general. 
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